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DETAILED ACTION 

Specification 

1 . The disclosure is objected to because of the following informalities: The Brief 
Description of the Drawings refers to figure 12 (page 3, line 28). The drawings only have 
1 1 figures and no reference is made to figure 1 1 in the Brief Description of the 
Drawings. It appears that the sentence in the specification actually is referring to figure 

1 1 and will be assumed as such for the remainder of this action. 
Appropriate correction is required. 

Claim Rejections - 35 USC § 103 

2. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

3. Claims 1-4 and 9 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Kotlikov et al. (US 2003/0012453 A1) in combination with Shimazu et al. 
(US005724454A). 

Regarding claim 1, Kotlikov teaches [a] method of detecting thin occlusion pixels 
in a digital image, comprising: 

identifying pixels from the digital color image representing one or more 
background/non-object regions ("classification of image data into object and non-object 
regions", Kotlikov, paragraph 25, line 2); 



Application/Control Number: 10/816,317 Page 3 

Art Unit: 2609 

detecting pixels representing object regions in the background/non-object regions 
("nonobject regions", Kotlikov, paragraph 25, line 3); and 

using the detected non-object pixels to replace the object pixels ("amending the 
object data to more closely resemble the data of non-object regions", Kotlikov, 
paragraph 24, line 1). 

While Kotlikov describes the objects to be replaced as a "scratch or other defect 
in the image" at paragraph 005, Kotlikov does not expressly teach that his non-object 
regions are sky regions and his object regions are hanging wire regions. 

Shimazu, working the same field of endeavor of color image correction, does 
describe replacing hanging wire pixels in sky regions ("elimination of a undesirable 
image area in an image, for example, an electric wire running across the blue sky", 
Shimazu, column 26, line 2. 

It would have been obvious at the time the invention was made to one of ordinary 
skill in the art to use the defect removal method of Kotlikov to solve the problem 
identified by Shimazu of removing the electric wire running across the blue sky in order 
to effect the "elimination of a undesirable image area in an image" (Shimazu, column 
26, line 2). 

Regarding claim 2, the Kotlikov-Shimazu combination teaches the elements 
common with claim 1 as described above. It also teaches developing a model based on 
the identified sky pixels, wherein such model is a mathematical function that has inputs 
of pixel position and outputs of color ("the pixel may be replaced by the average or 
weighted average of pixels in its neighborhood", Kotlikov, paragraph 50, line 3; and 
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using the model to operate on the digital color image to replace the values of pixels from 
the sky occlusion regions with values predicted by the model ("the pixel may be 
replaced by the average or weighted average of pixels in its neighborhood", Kotlikov, 
paragraph 50, line 2). 

Regarding claim 3, the Kotlikov-Shimazu combination teaches all the elements of 
claim 2 and further teaches wherein the model is a two-dimensional polynomial of the 
pixel position in the digital color image ("The defect may also be removed by 
interpolation such as with linear interpolation or quadratic interpolation", Kotlikov, 
paragraph 50, line 16). 

Regarding claim 4, the Kotlikov-Shimazu combination teaches [t]he method of 
claim 3 wherein the polynomial is a second-order polynomial ("The defect may also be 
removed by interpolation such as with linear interpolation or quadratic interpolation", 
Kotlikov, paragraph 50, line 16). 

Regarding claim 9, the Kotlikov-Shimazu combination teaches [a] method of 
removing hanging wire region pixels from detected sky regions in a digital color image 
having pixels, the method comprising: 

All the elements that are common with claim 1 are described in the claim 1 
section above. It also teaches: developing a model based on the identified sky pixels, 
wherein such model is a mathematical function that has inputs of pixel position and 
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outputs of color ("the pixel may be replaced by the average or weighted average of 
pixels in its neighborhood", Kotlikov, paragraph 50, line 3; and using the model to 
operate on the digital color image to replace the values of digital color image pixels 
associated with the hanging wire regions with values predicted by the model to thereby 
remove the hanging wire region pixels ("the pixel may be replaced by the average or 
weighted average of pixels in its neighborhood", Kotlikov, paragraph 50, line 3). 

4. Claim 5 is rejected under 35 U.S.C. 103(a) as being unpatentable over Kotlikov 
et al. (US 20030012453 A1) and Shimazu et al. (US005724454A) in combination with 
Luo et al. (US 20030053686A1 ). 

Regarding claim 5, the Kotlikov Shimazu combination teaches all the elements 
that are in common with claims 1 and 2. 

The Kotlikov-Shimazu combination does not teach (iii) using the model to operate 
on the digital color image to classify additional pixels not included in the initial sky region 
as sky. 

Luo, working in a similar problem solving area of color image segmentation, does 
teach using the model to operate on the digital color image to classify additional pixels 
not included in the initial sky region as sky ("an inventive region growing process is used 
to fill in holes and extend boundaries of the candidate cloudy sky regions." Luo '686, 
paragraph 36). 

It would have been obvious at the time the invention was made for one of 
ordinary skill in the art to further combine the combination of Kotlikov and Shimazu with 
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the region growing technique of Luo'686 in order to deal with regions where " 'marginal' 
pixels may have sky-color and texture belief values that barely fail the global threshold 
but are close enough to the belief values of the neighboring pixels that have passed the 
initial global threshold", Luo'686, paragraph 36. 

5. Claims 7 and 8 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Luo (US 200301 52289A1) and Miceli et al. (US 200301 281 49A1). 

Regarding claim 7, Luo teaches [a] method of determining the orientation of a 
digital color image ("a method for determining the orientation of an image", Luo'289, 
paragraph 1 ) having pixels, the method comprising: identifying pixels from the digital 
color image representing one or more sky regions (Luo'289, Fig. 1, reference 221); 
detecting one or more non-sky regions by examining the sky regions (Luo'289, Fig. 7, 
"Other subject matters"); and analyzing the non-sky regions to determine the orientation 
of the digital color image (Luo'289, Fig. 7, Orientation from Semantic Object). 

Luo'289 does not expressly teach using hanging wire regions to determine image 
orientation, but he does teach that "image orientation can be determined from a plurality 
of semantic objects, including human faces, sky, text, sign, grass, snow field, open 
water, or any other semantic objects that appear frequently in images, have strong 
orientations by themselves, have orientations strongly correlated with image orientation, 
and last but not least, can be detected with reasonably high accuracy automatically." 
(Luo'289, paragraph 28). 



Application/Control Number: 10/816,317 Page 7 

Art Unit: 2609 

Miceli, working in a similar problem solving area of analysis of overhead line 
geometries, teaches that overhead lines do have a strong orientation ("The simplest 
features of the geometrical configurations are illustrated with FIG. 1(c), which shows an 
overhead line 22 suspended between two poles 10. The weight of the overhead line 22 
causes it to sag in a characteristic catenary shape, with the maximal sag from the points 
of suspension occurring midway between the poles 10." Miceli, paragraph 39). 

It would have been obvious at the time the invention was made for one of 
ordinary skill in the art to apply the method of Luo to make use of the features of power 
lines, in particular including hanging wires with their characteristic catenary sag as 
described by Miceli, as the basis for a semantic object detector as described by Luo 
("... semantic object detectors can be used to produce alternative or additional 
estimates of the image orientation." Luo, paragraph 25). 

Such a modification would have been obvious to one of ordinary skill in the art 
because power lines are a ubiquitous feature of everyday urban (and even rural) 
landscapes ("The power-utility-system infrastructure alone in North America includes 
approximately 150,000,000 wooden pole structures used to support overhead lines." 
Miceli, paragraph 3). Power lines also have a strong gravitational orientation as 
described by Miceli, just as the road "signs" and other features described by Luo. That 
is, Luo states, "Text and signs appear in many pictures, e.g., street scenes, shops, etc. 
In general, it is unlikely that signs and text are placed sideways or upside down ..." at 
paragraph 006. In like manner, power lines have a definite orientation with respect to 
gravity, and it is impossible that they would be hung "up-side-down", making them even 



Application/Control Number: 10/816,317 Page 8 

Art Unit: 2609 

more reliable than road signs. Such a strong gravitational orientation coupled with their 
prevalence renders a power line a reliable feature for determining orientation. One of 
ordinary skill and creativity in the art would clearly recognize this, even in the absence 
of the teaching of Miceli. 

Regarding claim 8, the combination of Luo and Miceli teaches [t]he method of 
claim 7 wherein analyzing the hanging wire regions further comprises: determining the 
direction of gravity by examining the location of pixels of the hanging wire region with 
respect to the endpoints of the hanging wire region ("The weight of the overhead line 22 
causes it to sag in a characteristic catenary shape, with the maximal sag from the points 
of suspension occurring midway between the poles 10." Miceli, paragraph 39). 

6. Claim 6 is rejected under 35 U.S.C. 103(a) as being unpatentable over the 
combination of Kotlikov et al. (US 2003/0012453 A1) and Shimazu et al. 
(US005724454A) in combination with the combination of Luo (US 200301 52289A1) and 
Miceli et al. (US 200301 281 49A1). 

Regarding claim 6, the Kotlikov and Shimazu combination teaches the elements 
of claim 2 as described above. The combination of Kotlikov and Shimazu does not 
teach determining when sky occlusion regions are formed by hanging wires and 
determining the orientation of the image based on the detected hanging wire regions. 
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The combination of Luo'289 and Miceli does teach determining when 
sky occlusion regions are formed by hanging wires and determining the orientation of 
the image based on the detected hanging wire regions (see rejection above for claim 7). 

It would have been obvious at the time the invention was made for one of 
ordinary skill in the art to merge the hanging wire removal technique of the combination 
of the Kotlikov and Shimazu with the image orientation method of the Luo'289 and 
Miceli combination in order to avoid mis-oriented images since "it is aggravating if some 
of the images are displayed upside-down or sideways." (Luo'289, paragraph 3). 

Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Thomas M. Redding whose telephone number is (571) 
270-1579. The examiner can normally be reached on Mon - Fri 7:30 am - 5:00 pm EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Brian P. Werner can be reached on (571 ) 272-7401 . The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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